WHAT IS CLAIMED IS: 



1 . A computer-implemented process for segmenting writings written 
on a whiteboard work surface, comprising the process actions of: 

5 capturing images of a whiteboard that includes one or more items 

projected on the whiteboard and writings written on the whiteboard with a 
camera; and 

separating into two files the one or more items projected on the 
whiteboard and the writings written on the whiteboard. 

o 

2. The computer-implemented process of Claim 1 further comprising 
the process action of transmitting the file containing the writings written on the 
whiteboard to remote participants. 



15 3. The computer-implemented process of Claim 1 further comprising 

the process action of archiving the file containing the writings written on the 
whiteboard. 

4. The computer-implemented process of Claim 1 wherein the one or 
20 more items projected on the whiteboard comprises an electronic presentation. 

5. The computer-implemented process of Claim 1 wherein the one or 
more items projected on the whiteboard comprises annotations received from 
remote participants. 
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6. The computer-implemented process of Claim 1 wherein separating 
into two files the one or more items projected on the whiteboard and the writings 
written on the whiteboard comprises the process actions of: 

5 determining a relationship between the position of the camera view and 

the one or more items projected on the whiteboard via geometric calibration; 

determining a relationship between the colors of the writings written on the 

whiteboard and the colors of the writings written on the whiteboard captured in 

the images via color calibration; 
10 estimating a visual echo image that corrects a captured image to match 

the whiteboard in geometry and color using the results of the geometric 

calibration and the color calibration; 

using visual echo cancellation to isolate writings written on the whiteboard 

by comparing the ratio of the albedo of the captured image to the albedo of the 
15 visual echo image. 

7. The computer-implemented process of Claim 6 wherein the visual 

echo comprises annotations from remote participant and projected content. j 

i 

20 8. The computer-implemented process of Claim 6 wherein if the ratio 

of the albedo of the captured image to the albedo of the visual echo is less than 
1 , then a given pixel[x,y] belongs to the writings if and only if 
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1 -(albedo ratio for the sum of the red, green and blue color 
channels)/3 is greater than (the sum of the variances of the pixel for the red, 
green and blue color channels)/(the sum of the visual echoes for the pixel for the 
red, green and blue color channels). 

9. The computer-implemented process of Claim 8 wherein for a pixel 
[x,y] that belongs to the writings, for a given color channel, the color in the 
captured image can be corrected by multiplying its albedo ratio for the given 
channel by 255. 

10. A computer-implemented process for identifying content written on 
a whiteboard comprising the process actions of: 

obtaining geometric calibration data determining the geometric 
relationship between a whiteboard and a view of said whiteboard captured by a 
camera; 

obtaining color calibration data determining the color relationship 
between the color of writings written on the whiteboard and the color of writings 
written on a whiteboard in an image captured by said camera; 

capturing an image of said whiteboard having projected content 
and writings written on the whiteboard; 

calculating a visual echo of the captured image by correcting the 
color and geometry of the captured image using the geometric calibration data 
and the color calibration data; and 
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using the ratio of the albedo of the captured image to the albedo of 
the visual echo to identify writings on the whiteboard. 



1 1 . The computer-implemented process of Claim 1 0 further comprising 
5 the process actions of: 

segmenting out the identified writings; and 

transmitting the segmented writings to a remote site for display. 

1 2. The computer-implemented process of Claim 1 0 further comprising 
10 the process actions of: 

segmenting out the identified writings; and 
saving the segmented writings for later viewing. 



13. The computer-implemented process of Claim 10 wherein obtaining 
15 geometric calibration data comprises the process actions of: 

projecting a sequence of rectangular patterns onto the whiteboard 
and simultaneously capturing the sequence of rectangular patterns in a 
sequence of images; 

detecting the corners of the rectangular patterns in the sequence of 
20 images captured with the camera; and 

using the correspondences between the locations of the corners in 
the sequence of images and the corners of the rectangular patterns on the 
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whiteboard to estimate a homography that defines the geometric relationship 
between the camera view and the whiteboard. 

14. The computer-implemented process of Claim 13 wherein detecting 
5 the corners comprises the process actions of: 

converting the images to grayscale images; 
using edge detection to detect edges of the rectangles in the 
grayscale images thereby creating an edge map; 

using a Hough transform to detect straight lines on the edge map; 
10 fitting a quadrangle for each of the rectangles using the detected 

straight lines; and 

finding the corners of the quadrangle. 

15. The computer-implemented process of Claim 10 wherein obtaining 
15 the color calibration data comprises the process actions of: 

quantizing red, green, blue color space into bins; 
for each of the set of quantized colors, 

projecting a known color onto the full whiteboard that is 
divided into sub-regions; 
20 taking n images of said known color projected on the 

whiteboard with a camera and storing said n images; 

warping each of said n images into whiteboard coordinates 
using the homography found by the geometric calibration; 



-26- 



calculating the mean and variance of each color across n 
frames for each sub-region; and 

creating a lookup table of the mean and variance of each 
sub-region for said known color. 

16. The computer-implemented process of Claim 10 wherein if 
the ratio of the albedo of the captured image to the albedo of the visual echo is 
equal to 1 , then nothing is written on the whiteboard. 

1 7. The computer-implemented process of Claim 1 0 wherein if 
the ratio of the albedo of the captured image to the albedo of the visual echo is 
less than 1, then there are writings written on the whiteboard. 

18. The computer-implemented process of Claim 10 wherein if 
the ratio of the albedo of the captured image to the albedo of the visual echo is 
less than 1 , then a given pixel[x,y] belongs to the writings if and only if 

1 -(albedo ratio for the sum of the red, green and blue color 
channels)/3 is greater than (the sum of the variances of the pixel for the red, 
green and blue color channels)/(the sum of the visual echoes for the pixel for the 
red, green and blue color channels). 

19. The computer-implemented process of Claim 18 wherein for 
a pixel [x,y] that belongs to the writings, for a given color channel, the color in the 
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captured image can be corrected by multiplying its albedo ratio for the given 
channel by 255. 

20. A computer-readable medium having computer executable 
5 instructions for providing contents written on a whiteboard to remote meeting 
participants, said computer executable instructions comprising: 

obtaining geometric calibration data determining the geometric 
relationship between a whiteboard and a view of said whiteboard captured by a 
camera; 

10 obtaining color calibration data determining the color relationship 

between the color of writings written on the whiteboard and the color of writings 
written on a whiteboard and the color of the writings written on the whiteboard in 
an image captured by said camera; 

capturing an image of said whiteboard having projected content 
15 and writings written on the whiteboard; 

calculating a visual echo of the captured image by correcting the 
color and geometry of the captured image using the geometric calibration data 
and the color calibration data; and 

using the ratio of the albedo of the captured image to the albedo of 
20 the visual echo to identify writings on the whiteboard. 

21 . A system for segmenting a clear image of a whiteboard work 
surface for remote collaboration comprising: 
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a camera capturing one or more images of a whiteboard that includes one 
or more items projected on the whiteboard and writings written on the 
whiteboard; and 

a visual echo cancellation module for separating into two files the one or 
more items projected on the whiteboard and the writings written on the 
whiteboard. 

22. The system of Claim 21 further comprising: 

a remote site with a display for displaying the one or more items projected 
on the whiteboard and writings written on the whiteboard; and 

a network for sending said the one or more items projected on the 
whiteboard and writings written on the whiteboard to said remote site. 

23. The system on Claim 21 wherein the writings written on the 
whiteboard and the one or more items projected on the whiteboard are sent to 
the remote site in two separate files. 

24. The system of Claim 22 wherein the remote site has the capability 
to send back annotations to the whiteboard to be displayed with the one or more 
items projected on the whiteboard. 

25. The system of Claim 20 wherein the visual echo cancellation 
module for separating into two files the one or more items projected on the 
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whiteboard and the writings written on the whiteboard comprises sub-modules 
for: 

obtaining geometric calibration data determining the geometric 
relationship between a whiteboard and a view of said whiteboard captured by a 
5 camera; 

obtaining color calibration data determining the color relationship 
between the color of writings written on the whiteboard and the color of the 
writings written on the whiteboard in an image captured by said camera; 

capturing an image of said whiteboard having projected content 
10 and writings written on the whiteboard; 

calculating a visual echo of the captured image by correcting the 
color and geometry of the captured image using the geometric calibration data 
and the color calibration data; and 

using the ratio of the albedo of the captured image to the albedo of 
15 the visual echo to identify writings on the whiteboard. 

26. The system of Claim 25 wherein if the ratio of the albedo of 
the captured image to the albedo of the visual echo is less than 1 , then a given 
pixelfx.y] belongs to the writings if and only if: 
20 1 -(albedo ratio for the sum of the red, green and blue color 

channels)/3 is greater than (the sum of the variances of the pixel for the red, 
green and blue color channels)/(the sum of the visual echoes for the pixel for the 
red, green and blue color channels). 
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27. The system of Claim 26 wherein for a pixel [x,y] that belongs 
to the writings, for a given color channel, the color of the writings in the captured 
image is corrected by multiplying its albedo ratio for the given channel by 255. 

5 

28. The system of Claim 27 wherein for a pixel [x,y] that belongs 
to the writings, for a given color channel, the color in the captured image can be 
corrected by: 

choosing four classes of color corresponding to the most commonly 
10 used markers used to mark on the whiteboard; 

recovering the set of writings; 

correcting the color of the writings in the captured image for the red, 
green and blue color channels; 

converting the color of the writings from red, green and blue color 
15 space to hue, saturation and intensity color space to create a new image; 

manually defining writings corresponding to the four classes of 
color and histogramming these writings by hue, saturation and intensity; and 

evaluating the probability the each pixel in the new image belongs 
to one of the four classes based on the histogram. 

20 
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